Comments about the article in Nature: AI can learn to show its workings through trial and error

Following is a discussion about this article in Nature Vol 645 18 September 2025, by Daphne Ippolito & Yiming Zhang
To study the full text select this link: https://www.nature.com/articles/d41586-025-02703-7 In the last paragraph I explain my own opinion.

Reflection

Introduction

When a student encounters a challenging mathematics problem or a programmer needs to write a complex algorithm, they will rarely solve it all in one go.
When you want to solve a mathematical problem and you don't know the solution than you are better off to do that in smaller steps. That means you first try a simple example in order to get some insight.
When you want to write a complex program you are also better of to divide the whole in smaller tasks and specify each task as a subroutine.
Instead, they will reason through the task, jotting down notes and intermediate steps to arrive at a final solution.
This all makes sense if only humans are involved.
Likewise, large language models (LLMs) — artificial intelligence (AI) systems that process and generate human language — perform better at complex tasks when they write down their reasoning process before blurting out an answer than when they do not.
All AI systems operate on the basis: Text in, Text out.
The "Text in" is the (detailed) description of a problem in a human language. The "Text Out" describes the solution.
For example: "Text in": Write a computer program that decides if 77 is a prime number. Wikipedia: A prime number is a natural number greater than 1 that is not a product of two smaller natural numbers.
"Text Out" : The number 77 is the product of the numbers 7 and 11 and as such is not a prime number.
For the program
In a paper in Nature, the DeepSeek AI team2 reports that LLMs can be incentivized to learn to reason without ever being shown examples of human reasoning trajectories, using a trial-and-error process called reinforcement learning. So, what needs to be done to get an LLM to write out its reasoning process? Early efforts to elicit reasoning in LLMs simply added an extra instruction. Instead of prompting the LLM with “Q: Is 119 a prime number? A:” and expecting it to answer yes or no, researchers might input “Q: Is 119 prime? A: Let’s think step by step.” A small change in language was enough to induce the LLM to produce a step-by-step explanation — called a reasoning trace — before giving its answer.
This raises the question: Why does the LLM not always uses this step by step approach?
Other efforts taught LLMs to show their reasoning by presenting them with examples of humans using reasoning to solve problems. The LLM then learnt to produce reasoning traces that looked like the ones in the data — this is called supervised learning. However, prompting or training the LLM using human inputs can introduce biases, and these approaches prevent the model from developing its own ways of reasoning, which might perform better than human examples.
The researchers introduced a paradigm for eliciting reasoning steps from LLMs that are separate from the production of an answer. They implemented this in a model called DeepSeek-R1, which was released in January 2025. Rather than hoping that the LLM would reason when it was instructed to do so, or guiding it using examples of the human reasoning process, the researchers used a type of algorithm called reinforcement learning. Reinforcement-learning algorithms resemble how a child might learn to play a video game. As the child navigates their avatar through the game world, they learn through trial and error that some actions (such as collecting gold coins) earn points, whereas others (such as running into enemies) set their score back to zero. In a similar vein, DeepSeek-R1 was awarded a high score when it answered questions correctly and a low score when it gave wrong answers.
The researchers realized that, because maths and programming questions typically have verifiable answers, they could create a scoring system that helped the LLM to improve during the training process. The researchers’ main discovery was that, when the LLM was trained to produce correct answers using the trial-and-error process of reinforcement learning, it naturally learnt to output its reasoning (Fig. 1). This contrasts with previous prompting-based approaches, which were more akin to expecting a child to learn to master a video game by having them read the instructions, or supervised-learning approaches, which can be likened to expecting the child to master a game by watching a sibling play it hundreds of times.

Figure 1 | Reinforcement incentivizes large language models (LLMs) to write out their reasoning. 

LLMs are more effective at solving problems when they ‘reason’ by outputting the intermediate steps.
It does not make sense that the solutions are different if the only difference (of Reinforcement) is, that the intermediate steps are outputted or not.
The DeepSeek AI team2 reports that a training strategy called reinforcement learning can teach an LLM to reason without ever seeing examples of human reasoning.
Of course if reinforcement implies extra communication (teaching) with the application this can make a difference.
During training, the LLM was rewarded for correctly answering mathematical and programming questions and penalized for incorrect answers.
mathematical and programming applications are highly structured and put all the burden of asking the right question by the (intelligent) trainer.
The LLM learnt that reasoning improved the likelihood that it would produce the right answer, and it developed the ability to self-verify and self-reflect, enabling it to correct itself and check its performance before outputting an answer.
The most difficult part is: to correct itself . This raises the question to what extend "this learning, this reasoning" can be used for other applications.
Because it was trained using reinforcement learning, the LLM was not limited to learning human-defined reasoning patterns; it could also discover its own behaviours that earned high rewards.
More detail is required about what discovering means.
The researchers found that the LLM learnt to evaluate its own in-progress reasoning by reflecting on the statements it had already generated, and that it learnt to explore alternative approaches in its responses. As one example of this, the model learnt to insert phrases into its reasoning such as “Wait. That’s an aha moment I can flag here.” See: nature 18 September 2025 DeepSeek-R1-incentivizes-reasoning-in-LLMs-through-reinforcement-learning
However, the LLM also learnt certain behaviours which, although they might have helped it to produce better responses, resulted in reasoning traces that were difficult to understand. For example, the LLM adopted a behaviour in which its reasoning would switch back and forth between Chinese and English (the two languages the LLM was optimized to understand). The researchers also found that the LLM learnt to produce extremely long reasoning traces, which can contain 10,000 words or more. Furthermore, the reinforcement-learning method had to be trained on questions with clear-cut right or wrong answers (such as maths problems). This meant that the LLM didn’t learn how to handle questions requiring nuanced, subjective or long-form responses.
The researchers show that many of these issues were resolved by using a multistage training framework, in which the LLM was exposed to alternating stages of reinforcement learning and supervised learning. Trained in this way, DeepSeek-R1 achieved state-of-the-art accuracy on tasks that assessed maths and coding skills, factual knowledge and other forms of language understanding, in both Chinese and English.
Ultimately, the question of what makes a good reasoning LLM is a philosophical as much as a technical one.
Reasoning, processing all the information available, is a technical question. To evaluate, what defines (proper) reasoning, is something else.
What behaviours do users want from an AI when they ask it hard questions?
The same behaviour as the between humans., because that is in a manner i.e. language, they understand.
At one extreme, imagine an AI that has learnt to reason in a gibberish language that no human can hope to understand.
How is it possible to train a program using a language that no human understands?
More detail is required.
Should we care that its reasoning is completely unintelligible, so long as it arrives at the correct answer?
In that case the intermediate results can not be called (some sort of) reasoning.
What is important that when the answer is correct these intermediate results should be as expected. This is important that when the answer is wrong the intermediate results should give more detail what went wrong.
The version of DeepSeek-R1 that was trained through reinforcement learning alone tended to produce responses that were convoluted, long or otherwise difficult for humans to read. Ultimately, the researchers found that they needed to introduce some supervised learning to strike a balance between effective reasoning and intelligible responses to a broad variety of user queries.
DeepSeek-R1 has developed from a powerful but opaque solution-finder into a system that is capable of human-like conversations. This journey reflects the need for AI systems that not only accurately solve problems but are also tools that humans can understand, trust and meaningfully collaborate with.

1. Chalenges

Reflection 1 - Program to calculate prime numbers.

The purpose of this program to answer the question if a certain number is a prime number.
Input of the program is the number under investigation. Output is the answer yes or no.
The real issue is how does the AI system writes this program.
The

Reflection 2

If you want to give a comment you can use the following form Comment form

Created: 20 December 2024

Back to my home page Index
Back to Nature comments Nature Index